feat: scripts/cascade_for_combo.py — per-combo cascade stats - #48
Conversation
Adds a script that, given (benchmark, agent_model, mode), walks
runs/<benchmark>/, joins each annotation against its cloud manifest
(results/<benchmark>/cloud/<id>/manifest.json, with a GCS fallback that
caches locally), and prints the full cascade in two views:
* cumulative N1..N9 — monotone, with Δ vs prev (N1 is the headline
original-gold pass rate the user keeps asking for);
* mutually-exclusive partition L1..L11.
Two non-obvious rules baked in:
* latest-per-task uses annotated_at but SKIPS verdict=="eval_failed"
so a stale regrade whose grader infrastructure choked doesn't
override the genuine earlier verdict;
* model match is a case-insensitive substring (`opus` matches
`anthropic/claude-opus-4-7`) so the operator doesn't have to type
the full registry id.
The existing aggregate_cascading_latest in eval/cascading_report.py
applies neither filter, hence the new script rather than a flag.
Drive-by: fixes test_merge_no_overwrite_if_present, which was passing
only because a leftover runs/mini-interact/alien/alien_1/r1.json (from
another test's pollution) happened to exist. Now sandboxes BIRD_RUNS_ROOT
and pre-creates at the actual DEV-1533 dest path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 2 hours, 27 minutes, and 33 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds Changescascade_for_combo CLI and tests
Annotation destination path layout fix
Sequence Diagram(s)sequenceDiagram
participant CLI as main()
participant Collector as collect_latest_per_task
participant LoadManifest as load_manifest
participant LocalFS as Local FS (runs/)
participant GCS as GCS bucket
participant Aggregator as aggregate
participant Renderer as render
CLI->>Collector: benchmark, mode, agent_model
loop each annotation JSON
Collector->>LoadManifest: benchmark, run_id
LoadManifest->>LocalFS: read manifest.json
alt present locally
LocalFS-->>LoadManifest: dict
else allow_gcs=True
LoadManifest->>GCS: download blob
GCS-->>LoadManifest: bytes
else allow_gcs=False
LoadManifest-->>Collector: None (skip run)
end
LoadManifest-->>Collector: manifest dict
Collector->>Collector: filter by mode + model_matches
Collector->>Collector: group by (db, instance_id), pick latest non-eval_failed
end
Collector-->>CLI: chosen paths + counters
CLI->>Aggregator: chosen paths
Aggregator-->>CLI: n_counts + p_counts
CLI->>Renderer: benchmark, mode, agent_model, agg, counters
Renderer-->>CLI: formatted table string
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
…ed ADC main() eagerly built a google.cloud.storage.Client at startup, which raises DefaultCredentialsError on machines without ADC even when every needed manifest is already on disk under results/<benchmark>/cloud/<run_id>/. Drop the eager construction — gcs.read_manifest already lazily builds a client on first fallback, and load_manifest's except-Exception clause already swallows credential errors as "skip this run". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…invalid→eval_failed migration
Two adjacent correctness bugs in collect_latest_per_task:
(1) When every run for a task had verdict=eval_failed, the
`(real or recs)[-1]` fallback added the latest eval_failed row to
the chosen list, contradicting the docstring's "SKIPPING runs whose
evaluation.verdict == 'eval_failed'" contract and silently miscounting
grader-infrastructure failures as L11 hard fails. Now: omit
eval_failed-only tasks and record them under a new
`skipped_eval_failed_only` counter (also surfaced in the human render).
(2) SubmissionAnnotation._migrate_invalid_verdict upgrades legacy
`verdict="invalid"` + `failure_classification.primary="other"` to
`"eval_failed"` on read, but the selection logic read raw JSON and
only compared against the unmigrated string — so legacy infra
failures could be picked as the latest "real" verdict, defeating
the central skip rule. Mirror the migration in a small
`_effective_verdict` helper at the JSON-read boundary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ark-runs Reconciles this branch's DEV-1555 / DEV-1561 + CR r1 unification work with PR #46 (DEV-1550 SLayer compact-by-default), PR #48 (cascade_for_combo), PR #50 (query-syntax-rejection analysis), and DEV-1545 + DEV-1546 prompt additions on origin/main. Conflict resolutions: * ``_shared_otf_prompts.py`` — replaced HEAD's V1 helper section with origin/main's (it carries the new ``_DEDUP_VS_RAW_ROWS``, ``_TABLE_SET_PROBE``, ``_GRADER_ZERO_VS_ONE_DIAGNOSTIC``, ``_SLAYER_TOOLS_BLOCK`` etc. plus DEV-1550 ModelColumn / memory drill- in paragraphs), restored our ``_AFTER_REJECTED_DISCIPLINE`` (DEV-1555 stage-2), and kept our four ``*_V0`` snapshots appended at the bottom. Stripped lingering ``query_nested`` mentions from the V1 helpers so the unified-tool contract holds on both sides. * ``claude_sdk/agent.py`` — kept the unified ``query`` schema (``source_model`` + projection fields OR ``queries`` array; ``required: []``) and the runtime ``source_model XOR queries`` gate. The handler now builds the SlayerQuery JSON internally and forwards it to origin/main's DEV-1546 ``query_impl(query_json: str, …)`` for single-stage, or to ``query_nested_impl(queries=…)`` for the nested-DAG form — matching ``submit_query``'s pattern. Picked up origin/main's ``distinct_dimension_values`` field as an additional schema property so the DEV-1546 dim-only auto-dedup opt-out stays reachable through the unified shape. * ``eval/autopsy.py`` — combined origin/main's 2-attempt corrective- retry loop (sends Pydantic validation errors back as a ``tool_result`` so the model self-fixes the archeology_10 regression) with our DEV-1555 model-aware client (``_build_anthropic_client(model)`` + ``requires_thinking(model)`` thinking mode + auto tool_choice for Moonshot/Kimi) and the JSON-text fallback for third-party endpoints that don't honor forced tool_choice. The retry only fires when the model used the tool (the JSON-text-fallback path has no tool_use_id to bind the corrective ``tool_result`` to). * ``run.py._per_task_timeout_s`` — origin/main flipped the default to ``_DEFAULT_PER_TASK_TIMEOUT_S = 0.0`` (no cap). Adjusted our DEV-1555 grace logic so the runaway grace is only added when the operator explicitly opted in to a positive cap — the default-uncapped contract now holds. * V0 prompts kept as thin re-exports from ``_shared_otf_prompts.V0`` snapshots (origin/main had touched ``claude_sdk_otf*/prompts.py`` with DEV-1545/1550 helper composition, but our side reduced those files to one-line re-exports so the V0 snapshot is the single source of truth for the v0 surface). * ``test_shared_otf_prompts.py`` — re-baselined the V1 SHA pins to ``3fa05ac2…``/``65b8eb05…`` → final post-cleanse ``e671aea3…``/``a3fd695c…`` after stripping lingering ``query_nested`` mentions from the merged V1 helpers. * ``test_dev1534_query_wrapper.py`` — rewrote the schema pin to match the unified shape (``source_model``, ``queries``, ``required: []``); the prior DEV-1546 ``query_json``-only pin is superseded. * ``test_dev1546_distinct_dim_values.py`` — repointed the dedup composition tests at the ``SLAYER_OTF_*_V0`` snapshots (where the full ``_DEDUP_VS_RAW_ROWS`` body is inlined byte-for-byte); the v1 prompts teach the same guidance in a shorter form not assembled via the live constant. Rewrote the query-tool-schema test to assert the unified-shape surface. * ``test_dev1555_query_unified_schema.py`` — updated the mock ``query_impl`` to take ``query_json: str`` positional + kwargs (matches DEV-1546's signature) and parse it to verify the wrapper built the right SlayerQuery dict. Full non-integration suite: 3212 passed, 94 skipped, 50 deselected, 0 failed.
Summary
scripts/cascade_for_combo.py: given(--benchmark, --mode, --agent-model), walksruns/<benchmark>/<db>/<iid>/, joins each annotation against the cloud manifest (local cache → GCS fallback), picks the latest non-eval_failedrun per task, and prints BOTH the cumulative N1..N9 table (with Δ vs prev) and the mutually-exclusive L1..L11 partition.--jsonfor machine output;--no-gcsto disable the fallback.tests/cloud/test_fetch_annotation_merge.py::test_merge_no_overwrite_if_present, which was passing only because a leftoverruns/mini-interact/alien/alien_1/r1.json(from another test's pollution) happened to exist. Now sandboxesBIRD_RUNS_ROOTand pre-creates at the actual DEV-1533 dest path.CLAUDE.mdsection pointing at the script and naming its two non-obvious rules.Why a new script, not a flag on
aggregate_cascading_latestThe existing
aggregate_cascading_latestineval/cascading_report.pyfilters by neither(mode, agent_model)noreval_failed. Both filters matter:verdict=="eval_failed",primary=="other") and shouldn't override the genuine earlier verdict. Hits 36 raw + 11 slayer mini-interact tasks today.rglob+ manifest reads every time.Live numbers (post-fix)
mini-interact / opus / slayermini-interact / opus / rawTest plan
tests/scripts/test_cascade_for_combo.pypin: mode+model filter, eval_failed-override rule, missing-manifest-skip, partition aggregation.uv run python scripts/cascade_for_combo.py --benchmark mini-interact --mode slayer --agent-model opusreturns the table above.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests